Spectrally-truncated kernel ridge regression and its free lunch

Kernel ridge regression (KRR) is a well-known and popular nonparametric approach with many desirable properties, including minimax rate-optimality in estimating functions that belong to common reproducing kernel Hilbert spaces (RKHS). The approach, however, computationally intensive for large data sets, due the need operate on dense n×n matrix, where n sample size. Recently, various approximation schemes solving KRR have been considered, some analyzed. Some approaches such as Nyström sketching shown preserve rate optimality of KRR. In this paper, we consider simplest approximation, namely, spectrally truncating matrix its largest r<n eigenvalues. We derive an exact expression maximum risk truncated KRR, over unit ball RKHS. This result can be used study trade-off between level spectral truncation regularization parameter. show that, long RKHS infinite-dimensional, there threshold r, above which, spectrally-truncated surprisingly outperforms full terms risk, minimum taken strengthens existing results schemes, by showing not only one does lose rates, fact improve performance, all finite samples (above threshold). Moreover, implicit achieved substitute norm regularization. Both are needed achieve best performance.